Visual focus-aware techniques for visualizing display changes

ABSTRACT

A computer processor based method for controlling a plurality of computer displays in response to user behavior, the method comprising identifying at least one display that is unattended by the user by determining the user&#39;s visual focus, and applying an overlay window to the unattended display to control visualization of the identified unattended display.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This patent application is a continuation of and claims priority to U.S.Non-Provisional patent application Ser. No. 13/999,355, filed Feb. 12,2014, with the same title; the content of which is hereby expresslyincorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to visual focus-aware systems and methodsfor visualizing display changes in large display or multiple displaycomputing environments.

BACKGROUND OF THE INVENTION

Modern computer workstation setups regularly include multiple displaysin various configurations. With such multi-display setups more displayreal-estate is available than most are able to comfortably attend to.While the benefits of large or multi-display setups have beendemonstrated in several studies, it has also been suggested that thisincrease in display space leads to usability problems, window managementdifficulties and issues related to information overload.

Another potential issue is change blindness: users' inability to detectsignificant visual display changes when there is a disruption incontinuity such as a brief flicker or a shift in visual focus. Theeffects of change blindness in multi-display environments have not beenextensively studied. However, in one study, change blindness wasreported as being a significant factor for operators managing criticalevents using multi-display command and control systems with unattendeddisplays.

In general, the increased display real-estate afforded by multi-displaysetups means that users are unable to attend to all of it at once. Inparticular, this point is reached when the total display area is solarge that it does not fit within the user's field of vision. In thiscase, the user has to substantially turn their head to see differentparts of the display environment. This situation can arise when thenumber of displays or the distance between the displays increases. Forexample, it is likely to occur when users are working with threedisplays aligned bezel to bezel. When the user is only able to observepart of the multi-display environment, changes occurring on theunattended displays are difficult to track.

SUMMARY OF THE INVENTION

According to an aspect of the invention there is provided a computerprocessor based method for controlling a plurality of computer displaysin response to user behavior. The method involves identifying at leastone display that is unattended by the user by determining the visualfocus of the user and controlling visualization of the identifiedunattended display. By visual focus, it is meant a point or area inspace contained within a user's foveal field of vision. The unattendeddisplay is a display, which is not located within an area of visualfocus of a user.

Controlling visualization is done by applying an overlay window to anunattended display or window, thereby to control visualization of theidentified unattended display. An advantage of doing this is that thevisualization is generic and can be applied to any operating system orcomputer.

Controlling the visualization or look of unattended displays or screenscan help assist users in perceiving and tracking display changes inmulti-display environments or large display environments.

Controlling visualization of the unattended display may comprise causinga visual change indicative of the display being unattended.

Controlling visualization of the unattended display may comprisepresenting a static or fixed screen shot of contents of the displayprior to it becoming unattended and maintaining this on display untilthe display is next identified as being attended.

The method may involve monitoring updates associated with the unattendeddisplay. Monitoring updates may comprise capturing at least two screenframes and determining a difference between the two frames. A visualaspect of the unattended display may be changed in response to updatesassociated with the unattended display. For example, the brightness ofan area of the unattended display may be modified to show that an updatehas taken place, and/or a series of features may be presented, eachfeature being representative of an update. In the latter case, eachfeature may indicate a time of occurrence of the update. The series offeatures may be displayed around at least an area of the display.

Determining which display the user is looking at may be done bydetermining user's visual focus and using this to identify theunattended display.

According to another aspect of the invention, there is provided auser-interactive display management system comprising a plurality ofdisplays operable to display a plurality of windows, and a processoradapted to identify an unattended window by determining a user's visualfocus and to control appearance of the unattended window. The processormay be adapted to monitor updates associated with the unattended window.

A sensor or a monitor may be in communication with the processor, thesensor or monitor being operable to sense or monitor direction of visualfocus of a user.

According to yet another aspect of the invention there is provideduser-interactive display management system for implementing any of thecomputer controlled visualization processes described herein.

According to still another aspect of the invention there is provided acomputer program product have code or instructions for implementing anyof the computer controlled visualization processes described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the invention will now be described by way of exampleonly and with reference to the accompanying drawings, of which:

FIG. 1 is a front view of a user-interactive display management system;

FIG. 2 is a flow diagram of the stages of a first method for visualizingchanges in an unattended display;

FIG. 3 is a series of four screen frames as displayed using the methodof FIG. 2 between two successive display attendances;

FIG. 4 is a flow diagram of the stages of another method for visualizingchanges in an unattended display;

FIG. 5 is a series of four screen frames as displayed using one versionof the method of FIG. 4;

FIG. 6 is a detailed example of change visualized as bright pixels;

FIG. 7 is a series of four screen frames as displayed using anotherversion of the method of FIG. 4;

FIG. 8 is a detailed view of change visualized by brightening anapplication window;

FIG. 9 is a series of four screen frames as displayed using anotherversion of the method of FIG. 4; and

FIG. 10 is a detailed example of change visualized as rectangle ofvarying brightness projected around the application window.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a visual focus-aware user-interactive display managementsystem 10 that enables easy visualization of display changes onunattended displays. The system has a processor 12 in communication withtwo displays 14 and 16 adapted to show a plurality of applicationwindows 24 and 26, and a visualization application for controllingvisualization of the displays. In most implementations, the processor 12is part of a conventional PC and is operable to control normal PCfunctions, such as opening applications and displaying applicationwindows 24 and 26, as well as to control visualization of displaychanges on unattended displays.

Each display 14, 16 is equipped with a camera or sensor 18 a, 18 b incommunication with the processor 12. Each camera or sensor 18 a, 18 bmay be a conventional web camera that is attached to or is part of theassociated display. The cameras or sensors 18 a, 18 b together with theprocessor 12 form a visual focus monitor also referred to as adisplay-level visual focus detector for monitoring the visual focus 20of a user 22. When a display is located in the area of visual focus ofthe user 22 (i.e. an imaginary cone of space defined by the position ofthe user's eyes and the area in space visible within the user's fovealfield of view), it is referred to as the attended-display 14. When adisplay is not located in the area of visual focus of the user, it isreferred to as the unattended-display 16.

The processor 12 is configured to run a visual focus algorithm fortracking the visual focus 20 of the user 22 and a control displayprocess for controlling visualization of the displays 14 and 16. Gazedirection is a good approximation of visual focus. Techniques fordetermining direction of gaze are well known and so will not bedescribed in detail. Any technique known in the art could be used.

The visual focus-aware display management system controls the unattendeddisplay 16 to cause visualization of pixel level changes on theunattended display. Four different methods for doing this will bedescribed. These methods are intended to be used within an existing workcontext of the user. For this reason, the methods minimize userdistraction from his primary task. Each method visualizes displaychanges in a different manner associated with specific advantages interms of information content and distraction prevention. In use, theuser 22 might select the visualization method that suits best hisspecific requirements.

When the visualization application is first run, an overlay window isconstructed. The window is constructed only once and is then re-usedevery time visualization of an unattended display is active. When thewindow is shown, application focus is taken from the last activeapplication and given to the overlay window. This allows the overlaywindow to be positioned on top of all other windows or other visualelements on the display. When the overlay window is hidden, theapplication focus is given back to the last active application to avoidinterfering with user inputs. Using an overlay window to controlvisualization of an unattended display allows for deployment in anyoperating system or computer without modification.

The overlay window is created as follows. Firstly, a window is createdwith its content area exactly the same pixel resolution as the display.For example, if the display were 1024 pixels wide and 768 pixels high,the inner dimensions of the overlay window would also be 1024 pixelswide and 768 pixels high. Then all window decorations (window border,decorative frame, title bar buttons, title bar) are removed, so only thewindow content area is visible. Then the position of the overlay windowis translated, so that the top-left corner of the window is perfectlyaligned with the top-left corner of the display. This translationensures that the pixel coordinate systems of the window and the displayare aligned. This also means that every pixel position in the overlaywindow (and so in the visualizations) is shown directly on top of thecorresponding pixel on the display.

Once the size and position of the overlay window are finalized asdescribed above, the only remaining modifications are performeddepending on the visualization used and on interaction with the user.Both concern the opacity of the overlay window. The opacity of thewindow is controlled using the blending functions provided by thedeveloper or the operating system. The opacity values used range from 1to 0, where opacity of 1 means the window is fully opaque (i.e. only thepixels from the overlay window are visible), opacity of 0.5 means that50% of the color and brightness come from a pixel in the overlay windowand 50% come from the corresponding pixel on the underlying livedesktop, and opacity of 0 means that the overlay window is completelytransparent and only the live desktop pixels are visible.

FIG. 2 shows a flow diagram of the stages involved in a first method(method-one) for visualizing changes on an unattended display. Thishides visual changes occurring on an unattended display until the user'svisual focus shifts towards it again. When the user starts by attendinga display for example display 14 and switches visual focus from display14 to display 16, the visual focus monitor measures a shift in thevisual focus. This enables the processor to identify display 14 asunattended by the user 30. The processor then captures a frame ofdisplay 14 at the time the user shifted his visual focus to display 16.The captured frame is recorded and stored 32. The visualizationapplication uses the captured frame to generate an overlay window thatis a copy of the captured frame. This overlay image of the capturedframe overlays the display until the display becomes attended again. Theoverlay window is located in front of the real view of the unattendeddisplay, but does not interfere with that view, so that in the eventapplication windows are open, these continue to be up-dated or alteredas and when applications run in the background. The effect of this isthat when the overlay window is removed, the current state of theapplication windows is visible.

FIG. 3 shows a series of frames that illustrate method one. Frame A ofFIG. 3 shows an instant message chat window (specifically Google Talk).This is the last frame on display 14 before the user shifts visual focusaway from display 14. This frame is recreated using an overlay windowand is displayed 34 on display 14 as a black and white static,unchanging image, as shown in Frame B of FIG. 3. In this visualization,the overlay window has opacity of 1.0, meaning that the window is fullyopaque. Even if changes, such as a new instant message appearing in thechat window, occur at the application level, these are not displayedwhilst the display remains unattended. No visual change is then shown onthe unattended display 14 until the user switches visual focus fromdisplay 16 back to display 14 at which point the processor identifies 36the display 14 as attended. The processor then gradually, over a shorttime period, for example several seconds, changes 38 the opacity of thestatic black and white image (Frame B of FIG. 3) of the overlay windowon display 14 revealing the live display or application windows. Frame Cillustrates the gradual blending of the overlay window into the currentdisplay state. Frame D shows the current display state. Graduallychanging the opacity of the overlay window may happen regardless ofwhether a change has occurred on the unattended display. Alternatively,the opacity of the static-frame of the overlay window may changegradually only if a change has occurred on the unattended display.

FIG. 4 shows the stages involved in another method for visualizingchanges on an unattended display. Upon identification of the unattendeddisplay or window 40, the processor monitors updates on the unattendeddisplay or window 42 and changes a visual aspect of the display orwindow to reduce visual stimulus associated with the unattended window44, for example by overlaying a grey overlay window. The processor alsochanges a visual aspect of the unattended window in response to updatesassociated with the unattended display or window 46. This can be done ina number of ways, for example using temporal heat map visualizationmethods that show the user a level of change that occurred while thedisplay was unattended or visualizing short-term display changes on theunattended display. When the user returns his visual focus to thedisplay or window, the processor identifies the display or window asattended 48, and gradually changes its opacity to 0, revealing the livedisplay or application windows 50.

FIG. 5 shows a series of four screen frames, labeled A, B, C and Dobtained at various stages between two successive display attendancesusing a temporal heat map visualization method that shows the user thechanges that occurred while the display was unattended. Thisvisualization method will be referred to as method-two. When the usershifts his visual focus from display 14 to display 16, the processoridentifies display 14 as unattended 40. Frame A shows the last framebefore the user switches visual focus away from display 14. At thispoint, display 14 is darkened to reduce distractions 44. This is doneusing the overlay window with opacity set, for example, to 0.7. Frame Bshows a frame of the unattended display 14 before any visual changetakes place. While the user's visual focus is directed elsewhere, theprocessor monitors updates on display 14 by capturing a frame (ascreenshot of the contents of display 14) at regular intervals,approximately six times per second 42. The processor then computes adifference in pixel values between the two most recently captured framesfor the entire display and displays the change on display 14, 46 byvarying the brightness of the corresponding pixels in the overlaywindow. Frame C illustrates how the technique visualizes the change (anew instant message in the chat window) when it happens by brighteningthe pixels that changed in the chat window. This process continues untilthe user's visual focus returns to display 14, at which point theprocessor identifies display 14 as attended 48 and gradually changes theopacity of the image displayed on display 14 into a live image over aspecified timeframe 50 by gradually changing the overlay window'sopacity to 0 and then removing the window. Frame D shows the state ofdisplay 14 when the user returns his visual focus to display 14.

FIG. 6 shows an example of another unattended display on which twowindows are open and an overlay window is in place, together with anexpanded view of changes visualized as bright pixels. In this case, bothwindows have changed from the initial view and the opacity of theoverlay window is altered to brighten the pixels where changes aredetected.

The following formula is used to compute the value of pixels:Vnew=(Vprevious×Decay)+(Vdiff×Intensity), where Vnew is the new pixelvalue, Vprevious is the pixel value of the corresponding pixel in theprevious iteration of the heat map. Vdiff is the intensity of changebetween the last frame and the current frame measured as the differencein RGB values for the corresponding pixels. Decay is a fraction denotinghow quickly the current value should fade over time, and Intensitydefines how much of the intensity of Vdiff will be added to the heatmap. The decay and intensity are empirically determined parameters.

FIG. 7 shows a series of four screen snapshots labeled A, B, C and D,obtained using another temporal heat map visualization method. This willbe referred to as method-three. This visualization method worksidentically to method-two except that the amount of change in theunattended display is computed for application-window areas on thedisplay instead for individual pixels. Frame A shows the last framebefore the user shifts visual focus away from this display. Frame Bshows a frame of the unattended display before any visual change. FrameC illustrates how the technique visualizes change (a new instant messagein the chat window) as it happens by brightening the chat window. Inaddition, the content of the window is up-dated. Frame D shows thecurrent display state after the overlay window's opacity is reduced to 0and the overlay window is removed.

FIG. 8 shows a detailed example of change visualized by brightening theapplication window, where the change occurred. The decay and intensityparameters are empirically determined as described with reference tomethod-two. In this case, two windows are open in the last vieweddisplay, and the change is in the YouTube window. To visualize thischange, the opacity of the overlay window in the area corresponding tothe YouTube window is varied to brighten the YouTube window relative tothe other window. In this way, changes are visualized at the applicationwindow level.

FIG. 9 shows a series of four screen snapshots labeled A, B, C and Dobtained at various stages using another method for visualizing changeson an unattended display. This method uses short-term display changes tovisualize changes that occurred while the display was unattended. Thiswill be referred to as method-four. In this case, when the user shiftsvisual focus from display 14 to display 16, the processor identifiesdisplay 14 as unattended 40. Frame A shows the last frame before theuser shifts visual focus away from display 14. At this point, display 14is darkened by overlaying a suitably opaque overlay window to reducedistractions 44. Frame B shows a frame of the unattended display 14before any visual change takes place.

While the user's visual focus is directed elsewhere, the processormonitors updates on the unattended display 14 by continually capturingthe last twenty frames of the unattended display at approximatelyone-second intervals 42. The change for a particular frame is thenvisualized 46 as a thin rectangle around the window in which the changeoccurred and is visible on the unattended display 14. Frame Cillustrates a rectangle around a chat window, indicating occurrence of anew instant message. The brightness of the rectangle for eachapplication window is proportional to how much the window in the framechanged in relation to the previous frame. Since the processor tracksthe changes for the last twenty frames, the visualization ends up withtwenty evenly spaced thin rectangles around each window. The closer arectangle is to its window, the more recent the visualized change. Asbefore, the visualization is implemented using the overlay window, whichgenerates the rectangles as and when changes are detected.

FIG. 10 shows a detailed view of change visualized as rectangles ofvarying brightness projected around two application windows where changehas occurred. A window with many changes (such as a video player)results in cycles of bright rectangles, while a window with few changes(such as an instant messaging window) results in occasional brightrectangles. This process continues until the user's visual focus returnsto display 14, at which point the processor identifies display 14 asattended 48, following which the overlay window's opacity is graduallyreduced to 0 and the overlay window is removed to reveal the livedisplay or application windows, 50. Frame D shows the state of display14 when the user returns his visual focus to display 14.

A skilled person will appreciate that variations of the disclosedarrangements are possible without departing from the invention. Whilstthe invention is described primarily with reference to multipledisplays, it could also be used in a single display system. For example,the display as a whole could be monitored as described above. However,the visual focus detector tracks only whether the single display isattended or unattended. This requires no alteration of any part of thesystem. Alternatively, the method may be adapted for more fine grainedgaze tracking, where the single display is sub-divided into regions ofinterest (e.g. application windows if multiple applications share thedisplay space or even parts of an application window for visuallycomplex applications), which can be either attended or unattended. Inthis case, the visual focus and the corresponding point or area on adisplay has to be mapped to be able to evaluate which regions ofinterest are attended or unattended. Otherwise, the methods describedabove remain the same. Accordingly, the above description of thespecific embodiment is made by way of example only and not for thepurposes of limitation. It will be clear to the skilled person thatminor modifications may be made without significant changes to theoperation described.

The invention claimed is:
 1. A computer processor based method forcontrolling a plurality of computer displays in response to userbehavior, the method comprising: determining a user's visual focus usinga visual focus detector relative to the plurality of computer displays;identifying at least one display of the plurality of computer displaysthat is unattended by the user using the determined user's visual focus;generating, using a computer processor, an electronic overlay window onthe at least one identified unattended display; detecting updates to bedisplayed on the unattended display; and altering a visual aspect of theoverlay window on the unattended display in response to the detectedupdates so that the overlay window is displayed in an altered form onthe unattended display as to indicate to the user the updates on theunattended display.
 2. The method as claimed in claim 1, wherein theoverlay window causes a visual change indicative of the at least onedisplay being unattended.
 3. The method as claimed in claim 1, whereinthe overlay window is a static or fixed image of contents of the atleast one display prior to the at least one display becoming unattended.4. The method as claimed in claim 1 further comprising monitoringupdates associated with the at least one unattended display, whereinmonitoring updates comprises capturing at least two screen frames anddetermining a difference between the two screen frames.
 5. The method asclaimed in claim 1, wherein changing the visual aspect of the overlaywindow of the at least one unattended display further comprisesmodifying a level of brightness of an area of the at least oneunattended display to show that an update has taken place.
 6. The methodas claimed in claim 1, wherein changing the visual aspect furthercomprises displaying a series of features, each feature beingrepresentative of an update.
 7. The method as claimed in claim 1,wherein changing the visual aspect further comprises displaying a seriesof features, each feature being representative of an update andindicative of a time of occurrence of the update.
 8. The method asclaimed in claim 1, wherein changing the visual aspect comprisesdisplaying a series of features, each feature being representative of anupdate and the series of features being displayed around at least anarea of the at least one unattended display.
 9. The method as claimed inclaim 1, wherein the overlay window is adapted to modify a level ofbrightness of the at least one unattended display.
 10. Auser-interactive display management system comprising: a plurality ofdisplays operable to display a plurality of windows; and a processoradapted to: determine a user's visual focus using a visual focusdetector relative to the plurality of computer displays; identify atleast one display of the plurality of computer displays that isunattended by the user using the determined user's visual focus;generate an electronic overlay window on the unattended computerdisplay; detect updates to be displayed on the unattended computerdisplay; and alter a visual aspect of the overlay window on theunattended display in response to the detected updates so that theoverlay window is displayed in an altered form on the unattended displayas to indicate to the user the updates on the unattended display.
 11. Acomputer program product for causing one or more computer processors tocontrol a plurality of computer displays in response to user behavior,the computer program product comprising computer-readable programinstruction portions stored on a non-transitory computer-readablemedium, the computer program product program instruction portionscomprising: instruction portions for determining a user's visual focususing a visual focus detector relative to the plurality of computerdisplays; instruction portions for identifying at least one unattendeddisplay of the plurality of computer displays that is unattended by theuser using the determined user's visual focus; instruction portions forgenerating an electronic overlay window on the at least one identifiedunattended display; instruction portions for detecting updates to bedisplayed on the unattended display; and instruction portions foraltering a visual aspect of the overlay window on the unattended displayin response to the detected updates so that the overlay window isdisplayed in an altered form on the unattended display as to indicate tothe user the updates on the unattended display.
 12. The computer programproduct as claimed in claim 11, wherein the overlay window causes avisual change indicative of the at least one display being unattended.13. The computer program product as claimed in claim 11, wherein theoverlay window is a static or fixed image of contents of the at leastone display prior to the at least one display becoming unattended. 14.The computer program product as claimed in claim 11 further comprisinginstruction portions for monitoring updates associated with the at leastone unattended display, wherein monitoring updates comprises capturingat least two screen frames and determining a difference between the twoscreen frames.
 15. The computer program product as claimed in claim 11,wherein the instruction portions for changing the visual aspect of theoverlay window further comprise instruction portions for modifying alevel of brightness of an area of the at least one unattended display toshow that an update has taken place.
 16. The computer program product asclaimed in claim 11, wherein the instruction portions for changing thevisual aspect of the overlay window further comprise instructionportions for displaying a series of features, each feature beingrepresentative of an update.
 17. The computer program product as claimedin claim 11, wherein the instruction portions for changing a visualaspect of the overlay window further comprise instruction portions fordisplaying a series of features, each feature being representative of anupdate and indicative of a time of occurrence of the update.
 18. Thecomputer program product as claimed in claim 11, wherein the instructionportions for changing the visual aspect of the overlay window furthercomprise instruction portions for displaying a series of features, eachfeature being representative of an update and the series of featuresbeing displayed around at least an area of the at least one unattendeddisplay.
 19. The computer program product as claimed in claim 11 furthercomprising instruction portions for using the overlay window to modify alevel of brightness of the at least one unattended display.